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Abstract. This paper presents a novel technique for counterexample genera- 
tion in probabilistic model checking of Markov Chains and Markov Decision 
Processes. (Finite) paths in counterexamples are grouped together in witnesses 
that are likely to provide similar debugging information to the user. We list 
five properties that witnesses should satisfy in order to be useful as debugging 
aid: similarity, accuracy, originality, significance, and finiteness. Our witnesses 
contain paths that behave similar outside strongly connected components. 
This papers shows how to compute these witnesses by reducing the problem of 
I— I ' generating counterexamples for general properties over Markov Decision Pro- 

C/2 i cesses, in several steps, to the easy problem of generating counterexamples for 

O . reachability properties over acyclic Markov Chains. 
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1 Introduction 

Model checking is an automated technique that, given a finite-state model of a system 
and a property stated in an appropriate logical formalism, systematically checks the 
validity of this property. Model checking is a general approach and is applied in areas 
like hardware verification and software engineering. 

Nowadays, the interaction geometry of distributed systems and network protocols 
■ calls for probabilistic, or more generally, quantitative estimates of, e.g., performance 

and cost measures. Randomized algorithms are increasingly utilized to achieve high 
performance at the cost of obtaining correct answers only with high probability. For 
all this, there is a wide range of models and applications in computer science requir- 
ing quantitative analysis. Probabilistic model checking allow us to check whether or 
not a probabilistic property is satisfied in a given model, e.g., "Is every message sent 
successfully received with probability greater or equal than 0.99?". 

A major strength of model checking is the possibility of generating diagnostic infor- 
mation in case the property is violated. This diagnostic information is provided through 
a counterexample showing an execution of the model that invalidates the property under 
verification. Apart from the immediate feedback in model checking, counterexamples 
are also used in abstraction-refinement techniques |CGJ + 00| . and provide the founda- 
tions for schedule derivation (see, e.g., [BLR05] ) . 

Although counterexample generation was studied from the very beginning in most 
model checking techniques, this has not been the case for probabilistic model checking. 
Only recently attention was drawn to this subject [AHL05I AL06IHK07alHK07bl AL07j . 
fifteen years after the first studies on probabilistic model checking. Contrarily to other 
model checking techniques, counterexamples in this setting are not given by a single 
execution path. Instead, they are sets of executions of the system satisfying a certain 
undesired property whose probability mass is higher than a given bound. Since coun- 
terexamples arc used as a diagnostic tool, previous works on counterexamples have pre- 
sented them as sets of finite paths of large enough probability. We refer to these sets as 
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representative counterexamples. Elements of representative counterexamples with high 
probability have been considered the most informative since they contribute mostly to 
the property refutation. 

A challenge in counterexample generation for probabilistic model checking is that 
(1) representative counterexamples are very large (often infinite), (2) many of its el- 
ements have very low probability, and (3) that elements can be extremely similar to 
each other (consequently providing similar diagnostic information). Even worse, (4) 
sometimes the finite paths with highest probability do not indicate the most likely 
violation of the property under consideration. 

For example, look at the Markov chain M. in Figure [1] The property M. \= <o r 0^ 
stating that execution reaches a state satisfying ip (i.e., reaches S3 or S4) with~proba- 
bility lower or equal than 0.5 is violated (since the probability of reaching -0 is 1). The 
left hand side of table in Figure [5] lists finite paths reaching ip ranked according to their 
probability. Note that finite paths with highest probability take the left branch in the 
system, whereas the right branch in itself has higher probability, illustrating Problem 4. 
To adjust the model so that it does satisfy the property (bug fixing), it is not sufficient 
to modify the left hand side of the system alone; no matter how one changes the left 
hand side, the probability of reaching ip remains at least 0.6. Furthermore, the first 
six finite paths provide similar diagnostic information: they just make extra loops in 
s\. This is an example of Problem 3. Also, the probability of every single finite path 
is far below the bound 0.5, making it unclear if a particular path is important; see 
Problem 2 above. Finally, the (unique) counterexample for the property M. |= 0^ 
consists of infinitely many finite paths (namely all finite paths of A4); sec Problem 1. 
To overcome these problems, we partition a representative counterexample into sets of 
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Fig. 1: Markov chain 
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finite paths that follow a similar pattern. We call these sets witnesses. To ensure that 
witnesses provide valuable diagnostic information, we desire that the set of witnesses 
that form a counterexample satisfies several properties: two different witnesses should 
provide different diagnostic information (solving Problem 3) and elements of a single 
witness should provide similar diagnostic information, as a consequence witnesses have 
a high probability mass (solving Problems 2 and 4), and the number of witnesses of a 
representative counterexample should be finite (solving Problem 1). 

In our setting, witnesses consist of paths that behave the same outside strongly 
connected components. In the example of Figure [TJ there are two witnesses: the set of 
all finite paths going right, represented by [S0S2S4] whose probability (mass) is 0.6, and 
the set of all finite paths going left, represented by [S0S1S3] with probability (mass) 0.4. 

In this paper, we show how to obtain such sets of witnesses for bounded probabilistic 
LTL properties on Markov decision processes (MDP). In fact, we first show how to 
reduce this problem to finding witnesses for upper bounded probabilistic reachability 
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properties on discrete time Markov chains (MCs). The major technical matters lie on 
this last problem to which most of the paper is devoted. 

In a nutshell, the process to find witnesses for the violation of M. \= < ()ip, with M. 
being a MC, is as follows. We first eliminate from the original MC all the ^uninteresting" 
parts. This proceeds as the first steps of the model checking process: make absorbing 
all state satisfying ip, and all states that cannot reach ijj, obtaining a new MC M^. 
Next reduce this last MC to an acyclic MC hc(Mj.) in which all strongly connected 
components have been conveniently abstracted with a single probabilistic transition. 
The original and the acyclic MCs are related by a mapping that, to each finite path in 
Ac(A4^) (that we call rail), assigns a set of finite paths behaving similarly in M. (that 
we call torrent). This map preserves the probability of reaching ip and hence relates 
counterexamples in Ac(A4^) to counterexamples in M. Finally, counterexamples in 
Ac(A4^,) are computed by reducing the problem to a k shortest path problem, as in 
|HK07a| . Because Ac{M.^) is acyclic, the complexity is lower than the corresponding 
problem in |HK07aj . 

It is worth to mention that our technique can also be applied to simple pCTL 
formulas without nested path quantifiers. 

Organization of the paper. Section[2]presents the necessary background on Markov 
chains (MC), Markov Decision Processes (MDP), and Linear Temporal Logic (LTL). 
Section [3] presents the definition of counterexamples and discuss the reduction from 
general LTL formulas to upper bounded probabilistic reachability properties, and the 
extraction of the maximizing MC in a MDP. Section 2] discusses desire properties of 
counterexamples. In Sections O and [51 we introduce the fundamentals on rails and tor- 
rents, the reduction of the original MC to the acyclic one, and our notion of significant 
diagnostic counterexamples. Section [7] then present the techniques to actually compute 
counterexamples. In Section [8] we discuss related work and give final conclusions. 

2 Preliminaries 

2.1 Markov Decision Processes and Markov chains 

Markov Decision Processes (MDPs) constitute a formalism that combines nondetermin- 
istic and probabilistic choices. They are the dominant model in corporate finance, sup- 
ply chain optimization, system verification and optimization. There arc many slightly 
different variants of this formalism such as action-labeled MDPs [Bel57 FV97J, proba- 
bilistic automata [SL95ISdV04j : we work with the state-labeled MDPs from |BdA95] . 

Definition 2.1. Let 5 be a set. A discrete probability distribution on S is a function 
p: S — ► [0, 1] with countable or finite carrier and such that XLesP^) = 1- We- denote 
the set of all discrete probability distributions on S by Distr(S'). Additionally, we define 
the Dirac distribution on an element s £ S as l s , i.e., l s (s) = I and l s (t) = for all 
t eS\{s}. 

Definition 2.2. A Markov Decision Process (MDP) is a four-tuple V = (S, sq, L, t), 
where 

• S is the finite state space of the system; 

• so £ S is the initial state; 

• L is a labeling function that associates to each state sfSa set L(s) of propositional 
variables that are valid in s; 

• r: S — -> p(Distr(S*)) is a function that associates to each s £ S a non-empty and 
finite subset of Distr(S) of probability distributions. 

Definition 2.3. Let V = (S, so, r, L) be a MDP. We define a successor relation S C 
S x S by 5 = {(s, £) |3 7r S r(s) . n(t) > 0} and for each state s £ S we define the sets 

PathsfD, s) = {s siS2 ■■■ £ S u \s = s A Vn € N . 5(s n ,s n+1 )} and 

Paths* (V, s) = {sqSi . . . s„ £ S*\sq = s A V0 < i < n . S(s n , s n+ i)} 
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of paths and finite paths respectively beginning at s. We usually omit T> from the 
notation; we also abbreviate Paths(2?, sq) as Paths(2?) and Paths* (V, sq) as Paths* (2?). 
For id G Paths(s), we write the (n+l)-st state of lu as w„. As usual, we let B s C 
p(Paths(s)) be the Borel tr-algebra on the cones (so . . . s n ) = {oj G Paths(s)|wo = 
so A ... A uj n = s n }. Additionally for a set of finite paths A C Paths* (s), we define 

<4 = IU»- 




{"} {i'M [v] 



Fig. 3: Markov Decision Process 

Figure [3] shows a MDP. Absorbing states (i.e., states s with r(s) = {l s }) are 
represented by double lines. This MDP features a single nondctcrministic decision, to 
be made in state So, namely 7Ti and 7T2- 

Definition 2.4. Let D = (S,s ,t,L) be a MDP and A C 5. We define the sets of 
paths and finite paths reaching .A as 

Reach(2?, s, .A) = {ui G Paths(X>, s) | 3i>o-Wj G -4} and 
Reach* (P, s,A) = {a G Paths* (2?,s) | last(cr) G A A VK^i^.a, g ^4} 

respectively Note that Reach* (T>, s, A) consists of those finite paths a reaching A 
exactly once, at the end of the execution. It is easy to check that these sets are 'prefix 
free, i.e. contain finite paths such that none of them is a prefix of another one. 

2.2 Schedulers 

Schedulers (also called strategies, adversaries, or policies) resolve the nondeterministic 
choices in a MDP |PZ93IVar85IBdA95j . 

Definition 2.5. Let V = (S,so,t,L) be a MDP. A scheduler r\ on T> is a function 
from Paths*(X>) to Distr(p(Distr(S))) such that for all a G Paths* (V) we have rj(a) G 
Distr(r(last(<7))). We denote the set of all schedulers on T> by Sch(X>). 

Note that our schedulers arc randomized, i.e., in a finite path a a scheduler chooses 
an element of r(last(cr)) probabilistically. Under a scheduler 77, the probability that the 
next state reached after the path a is t, equals Sn-erfiastfo-)) ? 7( cr )( 7r ) ' 7r (^)- ^ n this way, 
a scheduler induces a probability measure on B s as usual. 

Definition 2.6. Let P be a MDP, s 6 S, and 77 an s-scheduler on T>. We define the 
probability measure n s _ n as the unique measure on B s such that for all sqSi . . . s n G 
Paths* (s) 

n-l 

Pr s ,,,((soSi...s„}) = J| r)(s s 1 ...Si)(iT)-Tr(si +1 ). 

i=0 7rer(si) 
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We now recall the notions of deterministic and memoryless schedulers. 

Definition 2.7. Let V be a MDP, s £ S, and 77 an scheduler of T>. We say that 77 is 
deterministic if 77(17) (7^) is either or 1 for all 7Tj £ r(last(<7)) and all a £ Paths* (T>) . 
We say that a scheduler is memoryless if for all finite paths <ti, o~i of 2? with last(cri) = 
last (02) we have 77(0-1) = 77(172) 

Definition 2.8. Let V be a MDP. s £ S, and Zi £ £> s . Then the maximal and minimal 
probabilities of A, Prj~(Z\), Pr~ (A), are defined by 

Prf(A)^ sup Pr a .„(4) and PrJ(A) 4 nf Pr (4). 

r/eSch s (X>) r/eSchsCD) 

A scheduler that attains Prj~(Z\) or Pr~(Z\) is called a maximizing or minimizing 
scheduler respectively. 

A Markov chain (MC) is a MDP associating exactly one probability distribution to 
each state. In this way nondeterministic choices are not longer allowed. 

Definition 2.9 (Markov chain). Let V = (S,s ,t,L) be a MDP. If \t(s)\ = 1 for all 
s £ S, then we say that V is a Markov chain (MC). 

2.3 Linear Temporal Logic 

Linear temporal logic (LTL) |MP91j is a modal temporal logic with modalities referring 
to time. In LTL is possible to encode formulas about the future of paths: a condition 
will eventually be true, a condition will be true until another fact becomes true, etc. 

Definition 2.10. LTL is built up from the set of propositional variables V, the logical 
connectives -1, A, and a temporal modal operator by the following grammar: 

::= V I -.0 I A I 4M<)>. 

Using these operators we define V, — (), and □ in the standard way. 

Definition 2.11. Let V = (S, sq, t, L) be a MDP. We define satisfiability for paths lo 
in V and LTL formulas <fi, t/j inductively by 

lo \= v v <f>v£ L(lo ) 

u hi, -«(> not(w hj, 0) 

u> \= 4> A t/' lo \= <t> and lo \= tp 

^ \=?, <^V> ^ 3i>o-^j Hj, i' and Vo<j«.w;j |= c 

where w^j is the i-th suffix of lo. When confusion is unlikely, we omit the subscript T> 
on the satisfiability relation. 

Definition 2.12. Let T> be a MDP. We define the language Sat x> (^>) associated to an 
LTL formula <j> as the set of paths satisfying 0, i.e. Sat I ,((/)) = {cj £ Paths(£>) | lo \= </>}. 
Here we also generally omit the subscript T>. 

Wc now define satisfiability of an LTL formula on a MDP V. Wc say that T> 
satisfies <f> with probability at most p (D \= <f>) if the probability of getting an execution 
satisfying <fi is at most p. 

Definition 2.13. Let V be a MDP, <j> an LTL formula and p £ [0, 1]. We define ^< 
and h> by 

2?h< p 0^Pr£(Sat(0))<p, 
2?|=~ p 0oPr;(Sat(0))>p. 
We define £> and X> \= >p <p m a similar way. 

In case the MDP is fully probabilistic, i.e., a MC, the satisfiability problem is 
reduced to V <^> Pr D (Sat(^)) XI p, where IX€ {<, <, >, >}. 
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3 Counterexamples 

In this section, we define what counterexamples are and how the problem of finding 
counterexamples for a general LTL property over Markov Decision Processes reduces 
to finding counterexamples to reachability problems over Markov chains. 

Definition 3.1 (Counterexamples). Let T> be a MDP and <j> an LTL formula. A coun- 
terexample to T> |= (f> is a measurable set C C Sat(0) such that PrJ(C) > p. Coun- 
terexamples to T> |= arc defined similarly. 

Counterexamples to 2? (= </> and 2? |= ^ cannot be defined straightforwardly as 
it is always possible to find a set C C Sat(</>) such that Pr^(C) < p or Pr^(C) < p, note 
that the empty set trivially satisfies it. Therefore, the best way to find counterexam- 
ples to lower bounded probabilities is to find counterexamples to the dual properties 
T> |= -Kp and T> |= <i That is, while for upper bounded probabilities, a counterex- 
ample is a set of paths satisfying the property beyond the bound, for lower bounded 
probabilities the counterexample is a set of paths that does not satisfy the property 
with sufficient probability. 

Example El Consider the MDP V of Figure g] 
and the LTL formula ()v. it is easy to check 
that V Qv. The set C = Sat(0«) = 
{ 7 e Path 1 s(s )|3 i >o.7 = s (si) l (s 4 H U { 7 £ 
Paths(so)|3i>o-7 = so(s3) l (s5)"} is a counterexam- 
ple. Note that Pr^, (C) = 1 where 77 is any deter- 
ministic scheduler of A4 satisfying T)(so) = tti. 

LTL formulas are actually checked by reduc- 
ing the model checking problem to a reachability 
problem dAKM97]. For checking upper bounded 
probabilities, the LTL formula is translated into 
an equivalent deterministic Rabin automaton and 
composed with the MDP under verification. On Fig- 4: 

the obtained MDP, the set of states forming accepting end components (maximal 
components that traps accepting conditions with probability 1) are identified. The 
maximum probability of the LTL property on the original MDP is the same as the 
maximum probability of reaching a state of an accepting end component in the final 
MDP. Hence, from now on we will focus on counterexamples to properties of the form 
T> \= or T> |= (}ip, where ip is a prepositional formula, i.e., a formula without 

temporal operators. 

In the following, it will be useful to identify the set of states in which a prepositional 
property is valid. 

Definition 3.2. Let V be a MDP. We define the state language Satx>(-0) associated to 
a prepositional formula ip as the set of states satisfying ip, i.e., Sat £>(?/>) = {s E S | s (= 
if,'}, where |= has the obvious satisfaction meaning for states. As usual, we generally 
omit the subscript T>. 

To find a counterexample to a property in a MDP with respect to a upper bound, 
it suffices to find a counterexample for the maximizing scheduler. A scheduler defines 
a Markov chain, and hence finding a counterexample on a MDP amounts to finding a 
counterexample in the Markov chain induced by the maximizing scheduler. The maxi- 
mizing scheduler turns out to be deterministic and mcmorylcss BdA95 ; consequently 
the induced Markov chain can be easily extracted from the MDP as follows. 

Definition 3.3. Let V = (S,Sq,t,L) be a MDP and r\ a deterministic memorylcss 
scheduler. Then we define the MC ^-associated to T> as T> v — (S, sq, V v , L) where 
V v (s,t) = (v(s))(t) for all s^teS. 




G 



Now we state that finding counterexamples for upper bounded probabilistic reach- 
ability LTL properties on MDPs can be reduced to rinding counterexamples for upper 
bounded probabilistic reachability LTL properties on MCs. 

Theorem 3.4. Let T> be a MDP, ip a propositional formula and p £ [0,1]. Then, 
there is a maximizing ( deterministic memoryless ) scheduler rj such that T> (= <=? 
V n ^= (}ip. Moreover, C is a counterexample to V v ^= tyip if and only if C is also a 
counterexample to T> \= ()ib. 

4 Representative Counterexamples, Partitions and Witnesses 

The notion of counterexample from Definition [XT] is very broad: just an arbitrary (mea- 
surable) set of paths with high enough probability. To be useful as a debugging tool (and 
in fact to be able to present the counterexample to a user), we need counterexamples 
with specific properties. We will partition counterexamples (or rather, representative 
counterexamples) in witnesses and list five properties that witnesses should satisfy. 

The first point to stress is that for reachability properties it is sufficient to consider 
counterexamples that consist of finite paths. 

Definition 4.1 (Representative counterexamples). Let T> be a MDP, ip a propositional 
formula and p £ [0,1]. A representative counterexample to V \= ()ip is a set C C 
Reach* (2?, Sat(i/>)) such that Pr^((C)) > p. We denote the set of all representative 
counterexamples to M. |=. (}ip by lZ(M,p,if>). 

Theorem 4.2. Let T> be a MDP, ip a propositional formula and p £ [0,1]. If C is a 

representative counterexample toT) \= ()ib, then (C) is a counterexample toT> \= (}ip. 

<p <p 

Furthermore, there exists a counterexample to T> \= ()ib if and only if there exists a 

<p 

representative counterexample to T> \= ip. 

<p 

Following I IKOTal . we present the notions of minimum counterexample, strongest 
evidence and most indicative counterexamples. 

Definition 4.3 (Minimum counterexample). Let M. be a MC, ip a propositional for- 
mula and p £ [0,1]. We say that C £ lZ(A4,p,ip) is a minimum counterexample if 
\C\ < |C'|, for all C £ n(M,p,ip). 

Definition 4.4 (Strongest evidence). Let A4 be a MC, ip a propositional formula and 
p £ [0, 1]. A strongest evidence to M. ^ 0-0 is a finite path a £ Reach*(A / (, Sat(^)) 
such that Pr M ((a)) > Pr M ((p)), for alTp £ Reach* (M, Sat(V>)). 

Definition 4.5 (Most indicative counterexample). Let Ai be a MC, ip a propositional 
formula and p £ [0, 1]. We call C £ r R.( y M.,p, ip) a most indicative counterexample if it is 
minimum and Pr((C)) > Pr((C')), for all minimum counterexamples C £ 1Z{M.,p,ili). 

Unfortunately, very often most indicative counterexamples are very large (even 
infinite), many of its elements have insignificant measure and elements can be extremely 
similar to each other (consequently providing the same diagnostic information). Even 
worse, sometimes the finite paths with highest probability do not exhibit the way in 
which the system accumulates higher probability to reach the undesired property (and 
consequently where an error occurs with higher probability). For these reasons, we 
are of the opinion that representative counterexamples are still too general in order 
to be useful as feedback information. We approach this problem by splitting out the 
representative counterexample into sets of finite paths following a "similarity" criteria 
(introduced in Section [5]). These sets are called witnesses of the counterexample. 

Recall that a set Y of nonempty sets is a partition of X if the elements of Y cover 
X and the elements of Y are pairwise disjoint. We define counterexample partitions in 
the following way. 
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Definition 4.6 (Counterexample partitions and witnesses). Let V be a MDP, ip a 

propositional formula, p € [0,1], and C a representative counterexample to D |= 

A counterexample partition Wc is a partition of C. We call the elements of Wc witnesses. 

Since not every partition generates useful witnesses (from the debugging perspec- 
tive), we now state properties that witnesses must satisfy in order to be valuable as 
diagnostic information. In Section [7] we show how to partition the detailed counterex- 
ample in order to obtain useful witnesses. 

Similarity: Elements of a witness should provide similar debugging information. 
Accuracy: Witnesses with higher probability should show evolution of the system 

with higher probability of containing errors. 
Originality: Different witnesses should provide different debugging information. 
Significance: The probability of a witnesses should be close to the probability bound 

P- 

Finiteness: The number of witnesses of a counterexamples partition should be finite. 
5 Rails and Torrents 

As argued before we consider that representative counterexamples are excessively gen- 
eral to be useful as feedback information. Therefore, we group finite paths of a repre- 
sentative counterexample in witnesses if they are "similar enough" . We will consider 
finite paths that behave the same outside SCCs of the system as providing similar 
feedback information. 

In order to formalize this idea, we first reduce the original Markov chain to an 
acyclic one that preserves reachability probabilities. We do so by removing all SCCs 
K of M. keeping just input states of K. In this way, we get a new acyclic MC denoted 
by Ac(A4). The probability matrix of the Markov chain relates input states of each 
SCC with its output states with the reachability probability between these states in 
M. Secondly, we establish a map between finite paths a in hc(M) (rails) and sets of 
finite paths W a in M (torrents). Each torrent contains finite paths that are similar, 
i.e., behave the same outside SCCs. Additionally we show that the probability of a is 
equal to the probability of W a . 

Reduction to Acyclic Markov Chains 

Consider a MC M. = (S, sq, V, L). Recall that a subset K C S is called strongly 
connected if for every s,ieK there is a finite path from s to t. Additionally K is called 
a strongly connected component (SCC) if it is a maximally (with respect to C) strongly 
connected subset of S. 

Note that every state is a member of exactly one SCC of M. (even those states that 
are not involved in cycles, since the trivial finite path s connects s to itself). From now 
on we let SCC* be the set of non trivial strongly connected components of a MC, i.e., 
those composed of more than one state. 

A Markov chain is called acyclic if it does not have non trivial SCCs. Note that an 
acyclic Markov chain still has absorbing states. 

Definition 5.1. Let M = (S,s ,V,L) be a MC. Then, for each SCC* K of M, wc 
define the sets Inp K C S of all states in K that have an incoming transition from a 
state outside of K and OutK C S of all states outside of K that have an incoming 
transition from a state of K in the following way 



Inp K 4{ u£ K|3seS\K .V(s, u) > 0}, 
Out K = {s £ S\K | 3ii G K .V(u, s) > 0}. 




• • • 



Output States 



We also define for each SCC* K a MC related to K as M K = (K U Out K , s K , Vk, L k ) 
where sk is any state in Inp K , Lk(s) = L(s), and VK(s,t) is equal to V(s,t) if s 6 K 
and equal to l s otherwise. Additionally, for every state s involved in non trivial SCCs 
we define SCC+ as Mk, where K is the SCC* of M such that s e K. 

Now we are able to define an acyclic MC Ac(yVf) related to M. 

Definition 5.2. Let M = (S,s ,P,L) be a MC. We define Ac(M) = (S', s ,V, L') 
where g s inp 



S'=S\ |J KU |J Iq Pk 



KgSCC* 



K6SCC* 



• V 



V'(s,t) 



'V(s,t) ifseS :o in • 

Pr A/is (Reach(SCC+,s,{t})) if s e S inp At e Out scc +, 

Is 





if s e Si n p A Out scc + 
otherwise. 



Note that Ac(Ai) is indeed acyclic. 



Example 2. Consider the MC Ai of Figure 5(a) The strongly connected components 
of M are Ki = {si, s 3 , s 4 , s 7 }, K 2 = {s5,S6,s 8 } and the singletons {so}, {S2}, {^9}, 
{sio}j { s ii}j { s i2}, {S13}! an d {S14}. The input states of Ki are Inp Kl = {si} and its 
output states are Outi^ = {sg, sio}. For K2, Inp K ? = { 55, sq} and OutK 2 = {sn,si4}. 
The reduced acyclic MC of M. is shown in Figure 5(b) 




(a) Original MC 



(b) Derived Acyclic MC 



Fig. 5: 
Rails and Torrents 

We now relate (finite) paths in Ac(A^) (rails) to sets of (finite) paths in Ai (torrents). 

Definition 5.3 (Rails). Let M be a MC. A finite path a E Paths* (Ac(M)) will be 
called a rail of M.. 

Consider a rail cr, i.e., a finite path of Ac(.M). We will use a to represent those 
paths co of M. that behave "similar to" a outside SCCs of A4 . Naively, this means that 
a is a subsequence of ui. There are two technical subtleties to deal with: every input 
state in a must be the first state in its SCC in co (freshness) and every SCC visited by uj 
must be also visited by a (inertia) (see Definition 15. 5p . We need these extra conditions 
to make sure that no path uj behaves "similar to" two distinct rails (see Lemma 15.71) . 

Recall that given a finite sequence a and a (possible infinite) sequence w, we say that 
cr is a subsequence of cj, denoted by a C cu, if and only if there exists a strictly increasing 
function / : {0, 1, . . . , \a\ — 1} — > {0, 1, . . . , \u>\ — 1} such that Vo<i<|cr|.c, = If w is 

an infinite sequence, we interpret the codomain of / as N. In case / is such a function 
we write a Qf uj. Note that finite paths and paths arc sequences. 



9 



Definition 5.4. Let Ai = (S,So,V,L) be a MC. On S we consider the equivalence 
relation satisfying s ^ t if and only if s and t are in the same strongly connected 
component. Again, we usually omit the subscript A4 from the notation. 

The following definition refines the notion of subsequence, taking care of the two 
technical subtleties noted above. 

Definition 5.5. Let Ai = (S, sq,V,L) be a MC, u> a (finite) path of Ai, and a G 
Paths* (Ac(Ai)) a finite path of Ac(Ai). Then we write a < oj if there exists / : 
{0, 1, . . . , |er| — 1} — > N such that a Cy u> and for all < i < \a\ we have 

V <j</(i) : w f(i) 7^ w j\ f° r all i = 0, 1, . . . \<j\ — 1, [Freshness property] 
V/(i)<j</(j+i) : Uf(i) ~ Wj', for all i = 0, 1, . . . \a\ — 2. [Inertia property] 

In case / is such a function we write a <f u. 



Example 3. Let = (S,so,V,L) be the MC of Figure 5(a) and take a = S0S2SGS14. 
Then for all i 6 N we have a where = sqS2S§{s^s%s§) % and /i(0) = 0, 

fi(l) = 1, /i(2) = 2, and /i(3) = 3 + 3i. Additionally, a ^ sqS<iSsS%s§S\± since for 
all / satisfying a Cy SoS2S5S8 s 6Si4 we must have /(2) = 5; this implies that / does 
not satisfy the freshness property. Finally, note that a ^ S0S2S6S11S14 since for all / 
satisfying a Qf S0S2S6S11S14 we must have /(2) = 2; this implies that / does not satisfy 
the inertia property. 

We now give the formal definition of torrents. 

Definition 5.6 (Torrents). Let M. = (S, so, V, L) be a MC and a a sequence of states 
in S. We define the function Torr by 

Torr(M, a) = {ui e Paths(A4) | a < lo}. 

We call Torr(A^,cr) the torrent associated to a. 

We now show that torrents are disjoint (Xemma l5.7p and that the probability of a 
rail is equal to the probability of its associated torrent (Theorem I5.10[) . For this last 
result, we first show that torrents can be represented as the disjoint union of cones of 
finite paths. We call these finite paths generators of the torrent (Definition 15.81) . 

Lemma 5.7. Let M be a MC. For every a, p e Paths* (Ac(M)) we have 

Torr(7W, a) n Torr(A<f, p) = 

Definition 5.8 (Torrent Generators). Let M. be a MC. Then we define for every rail 
a £ Paths* (Ac (M)) the set 

GenTorr(M, a) = {p e Paths* (M) \ 3f : a -< f p A f{\a\ - 1) = \p\ - 1}. 

In the example from the Introduction (see Figure [T|), S0S1S3 and S0S2S4 are rails. 
The associated torrents are, respectively, {sos"s3 | n G N*} and {so s 2 s 4 | « G N*} 
(note that S3 and S4 are absorbing states), i.e. the paths going left and the paths going 
right. The generators of the first torrent are {soS™S3 | n G N*} and similarly for the 
second torrent. 

Lemma 5.9. Let A4 be a MC and a G Paths* (Ac (A4)) a rail of Ai. Then we have 

Torr(X,a)= (+j (p). 

p6GenTorr(A^,(r) 

Theorem 5.10. Let M be a MC. Then for every rail a G Paths* (Ac (M)) we have 

Pr Ac ^ ) (^))= Pr J v 1 ( TOTr (-M^))- 
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6 Significant Diagnostic Counterexamples 

So far we have formalized the notion of paths behaving similarly (i.e., behaving the same 
outside SCCs) in a MC Ai by removing all SCC of Ai, obtaining Ac(A4). A representa- 
tive counterexample to Ac(A^) ^ ()tp will give rise to a representative counterexample 
to Ai \= ()ip. For every finite p~ath a in the counterexample to Ac(A'f) (= the 
set GenTorr(Af, a) will be a witness. The union of these is the representative coun- 
terexample to Ai 1= ()ib. 

<p 

Before giving a formal definition, there is still one technical issue to resolve: we need 
to be sure that by removing SCCs we are not discarding useful information. Because 
torrents are built from rails, we need to make sure that when we discard SCCs, we do 
not discard rails that reach ip. 

We achieve this by first making states satisfying ip absorbing. Additionally, we make 
absorbing states from which it is not possible to reach ip. Note that this does not affect 
counterexamples . 

Definition 6.1. Let Ai = (S, s^V, L) be a MC and ip a prepositional formula. We 
define the MC Ai^ = (S, s , V^, L), with 

{1 if s g Sat O) A s = t, 
1 if s G Sat(V>) As = t, 
V(s,t) if seSat (V>)-Sat(V>), 
otherwise, 

where Sat (?/>) — {s G S \ Pr s (Reach(A4, s, Sat(-0))) > 0} is the set of states reaching 
tp in Ai. 

The following theorem shows the relation between paths, finite paths, and prob- 
abilities of Ai, Ai^p, and Ac(.M^). Most importantly, the probability of a rail a (in 
Ac(A^^)) is equal to the probability of its associated torrent (in Ai) (item [5] below) 
and the probability of ()tp is not affected by reducing Ai to Ac(M^,) (item [6] below). 

Note that a rail a is always a finite path in Ac(A^^), but that we can talk about its 
associated torrent Torr(7W^, er) in Ai^ and about its associated torrent Torr(A / (, a) in 
Ai. The former exists for technical convenience; it is the latter that we are ultimately 
interested in. The following theorem also shows that for our purposes, viz. the definition 
of the generators of the torrent and the probability of the torrent, there is no difference 
(items |3] and 0] below) . 

Theorem 6.2. Let Ai = (S, sq,V, L) be a MC and ip a propositional formula. Then 
for every a G Paths* (AI,/,) 

1. Reach* (M$, s , Sat(V>)) = Reach* (M, so, Sat(tp)), 

2. Pr^((a» = Pr M ((,)), 

3. GenTorrO^, a) = GenTorr (M, a), 

4. Pr (Torr(AV^)) = P* M (Torr(A4, a)), 

6. Ac(M^,) (=, tyij) if and only if M |=, C"ip, for any p G [0, 1]. 

Definition 6.3 (Torrent-Counterexamples). Let Ai = (S, sq,V, L) be a MC, ip a 
propositional formula, and p G [0, 11 such that Ai y= §ib. Let C be a representative 
counterexample to Ac(A / l^,) \= < ()ip. We define the set 

TorRepCount(C) = {GcnTorr(Af, a) \ a G C}. 

We call the set TorRcpCount(C) a torrent- counterexample of C. Note that this set is a 
partition of a counterexample to Ai ^= ()ip. Additionally, we denote by lZt(Ai,p,ip) 
to the set of all torrent-counterexamples to Ai \= §ip, i.e., {TorRcpCount(C) | C G 
1l(M,p,i>)}. 
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Theorem 6.4. Let Ai = (S, So> V, L) be a MC, ip a propositional formula, and p G 
[0,1] such that Ai ^= ()ip. Take C a representative counterexample to Ac(A^^,) \= < 00- 
Then the set of finite paths l±JweTorRcpCount(c) ^ * s a representative counterexample to 

Ai K 0^. 

< P 

Note that for each a G C we get a witness GcnTorr( Ai, cr). Also note that the 
number of rails is finite, so there are also only finitely many witnesses. 

Following [H K07a| . we extend the notions of minimum counterexamples, strongest 
evidence and smallest counterexample to torrents. 

Definition 6.5 (Minimum torrent-counterexample). Let Ai be a MC, ip a proposi- 
tional formula and p G [0,1]. We say that Ct £ lZt(Ai,p,ip) is a minimum torrent- 
counterexample if |Ct| < \C' t \, for all C' t G TZt(A4,p,ip). 

Definition 6.6 (Strongest torrent-evidence). Let Ai be a MC, ip a propositional for- 
mula and p G [0,1]. A strongest torrent- evidence to Ai \f= 0-0 is a torrent W a G 
Torr(A4,Sat(0)) such that Pr M (W a ) > Pr M {W p ) for all Wp G Torr(.M, Sat(?/>))- 

Now we define our notion of significant diagnostic counterexamples. It is the gen- 
eralization of most indicative counterexample from [HK07aj to our setting. 

Definition 6.7 (Most indicative torrent-counterexample). Let Ai be a MC, i/j a propo- 
sitional formula and p G [0,1]. We call Ct G TZt(M,P,ip) a most indicative torrent- 
counterexample if it is a minimum torrent-counterexample and Pr(lJ M/gCt (W)) — 
P r (Uwgc; (^0) f° r au minimum torrent counterexamples Cj G lZ t (M,p,ip). 

By Theorem [63] it is possible to obtain strongest torrent-evidence and most indica- 
tive torrent-counterexamples of a MC Ai by obtaining strongest evidence and most 
indicative counterexamples of Ac(A'l^) respectively 

7 Computing Counterexamples 

In this section wc show how to compute most indicative torrent-counterexamples. We 
also discuss what information to present to the user: how to present witnesses and how 
to deal with overly large strongly connected components. 

7.1 Maximizing Schedulers 

The calculation of a maximal probability on a reachability problem can be performed 
by solving a linear minimization problem |BdA95ldA97j . This minimization problem 
is defined on a system of inequalities that has a variable Xi for each different state Si 
and an inequality Y]j it(sj) ■ Xj < xi for each distribution 7r G r(si). The maximizing 
(deterministic memoryless) scheduler 77 can be easily extracted out of such system 
of inequalities after obtaining the solution. If po, . . . , p n are the values that minimize 
J^Xj in the previous system, then 77 is such that, for all Si, 77(3,) = tt whenever 
J2j ■ pj = Pi- In the following we denote P s . = x;. 

7.2 Computing most indicative torrent-counterexamples 

Wc divide the computation of most indicative torrent-counterexamples to T> \= 00 

<p 

in three stages: pre-processing, SCC analysis, and searching. 

Pre-processing stage. We first modify the original MC Ai by making all states in 
Sat(-0) U S \ Sat (ip) absorbing. In this way we obtain the MC Ai^ from Definition 16. II 
Note that we do not have to spend additional computational resources to compute 
this set, since Sat o (-0) = {s G S P s . [ip] > 0} and hence all required data is already 
available from the LTL model checking phase. 
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SCC analysis stage. We remove all SCCs K of M.^ keeping just input states of K, 
getting the acyclic MC Ac(M^) according to Definition 15.21 

To compute this, we first need to find the SCCs of Ai^. There exists well known 
algorithms to achieve this: Kosaraju's, Tarjan's, Gabow's algorithms (among others). 
We also have to compute the reachability probability from input states to output states 
of every SCC. This can be done by using steady state analysis techniques |Cas93j . 

Searching stage. To find most indicative torrent-counterexamples in A4, we find 
most indicative counterexamples in Ac(Af^). For this we use the same approach as 
[HK07a| . turning the MC into a weighted digraph to exchange the problem of finding 
the finite path with highest probability by a shortest path problem. The nodes of the 
digraph are the states of the MC and there is an edge between s and t if V(s,t) > 0. 
The weight of such an edge is — log "P(s, t). 

Finding the most indicative counterexample in Ac(A4^,) is now reduced to finding 
k shortest paths. As explained in [HK07aj . our algorithm has to compute k on the 
fly. Eppstein's algorithm |Epp98| produces the k shortest paths in general in 0(m + 
nlogn + fc), where m is the number of nodes and n the number of edges. In our case, 
since Ac(A4^) is acyclic, the complexity decreases to 0(m + k). 

7.3 Debugging issues 

Representative finite paths. What we have computed so far is a most indicative 
counterexample to Ac(A^^,) (= (}tp. This is a finite set of rails, i.e., a finite set of paths 
in Ac(A4^,). Each of these paths a represents a witness GenTorr(A / J, cr). Note that this 
witness itself has usually infinitely many elements. 

In practice, one somehow has to display a witness to the user. The obvious way 
would be to show the user the rail a. This, however, may be confusing to the user as a 
is not a finite path of the original Markov Decision Process. Instead of presenting the 
user with cr, we therefore show the user the element of GenTorr(A / (, a) with highest 
probability. 

Definition 7.1. Let M be a MC, and a e Paths* (Ac(A4 (/ j)) a rail of M. We define 
the representant of Torr(A / l,cr) as 



repTorr (M , a) = repTorr l+l (p) = arg max Pr((p)) 

I w / pGGcnTorr(A1,cr) 

\pGGenTorr(A1,o-) / 

Note that given repTorr (A4, cr), one can easily recover a. Therefore, no information 
is lost by presenting torrents as a single element of the torrent instead of as a rail. 

Expanding SCC. It is possible that the system contains some very large strongly 
connected components. In that case, a single witness could have a very large probability 
mass and one could argue that the information presented to the user is not detailed 
enough. For instance, consider the Markov chain of Figure [6] in which there is a single 
large SCC with input state t and output state u. 

The most-indicative torrent counterexample to the 
property M. \= g (>ip is simply {GenTorr(siu)}, i.e., a sin- 
gle witness with probability mass 1 associated to the rail 
stu. Although this may seem uninformativc, wc argue that 
it is more informative than listing several paths of the form 
st ■ ■ ■ u with probability summing up to, say, 0.91. Our 
single witness counterexample suggests that the outgoing 
edge to a state not reaching ip was simply forgotten; the 
listing of paths still allows the possibility that one of the 
probabilities in the whole system is simply wrong. 
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Nevertheless, if the user needs more information to tackle bugs inside strongly 
connected components, note that there is more information available at this point. In 
particular, for every strongly connected component K, every input state s of K (even 
for every state in K), and every output state t of K, the probability of reaching t from 
s is already available from the computation of Ac(A4^) during the SCC analysis stage 
of Section O 

8 Final Discussion 

We have presented a novel technique for representing and computing counterexamples 
for nondeterministic and probabilistic systems. We partition a counterexample in wit- 
nesses and state five properties that we believe good witnesses should satisfy in order 
to be useful as debugging tool: (similarity) elements of a witness should provide similar 
debugging information; (originality) different witnesses should provide different debug- 
ging information; (accuracy) witnesses with higher probability should indicate system 
behavior more likely to contain errors; (significance) probability of a witness should 
be relatively high; (finitencss) there should be finitely many witnesses. We achieve this 
by grouping finite paths in a counterexample together in a witness if they behave the 
same outside the strongly connected components. 

Presently, some work has been done on counterexample generation techniques for 
different variants of probabilistic models (Discrete Markov chains and Continues Markov 
chains) AHL05 AL06 HK07a HK07b|. In our terminology, these works consider wit- 
nesses consisting of a single finite path. We have already discussed in the Introduction 
that the single path approach does not meet the properties of accuracy, originality, 
significance, and finitencss. 

Instead, our witness/torrent approach provides a high level of abstraction of a coun- 
terexample. By grouping together finite paths that behave the same outside strongly 
connected components in a single witness, we can achieve these properties to a higher 
extent. Behaving the same outside strongly connected components is a reasonable way 
of formalizing the concept of providing similar debugging information. This grouping 
also makes witnesses significantly different form each other: each witness comes form 
a different rail and each rail provides a different way to reach the undesired property. 
Then each witness provides original information. Of course, our witnesses are more sig- 
nificant than single finite paths, because they are sets of finite paths. This also gives us 
more accuracy than the approach with single finite paths, as a collection of finite paths 
behaving the same and reaching an undesired condition with high probability is more 
likely to show how the system reaches this condition than just a single path. Finally, 
because there is a finite number of rails, there is also a finite number of witnesses. 

Another key difference of our work to previous ones is that our technique allows us 
to generate counterexamples for probabilistic systems with nondctcrminism. However, a 
recent report [AL07] also considers counterexample generation for MDPs. This work is 
limited to upper bounded pCTL formulas without nested temporal operators. Besides, 
their technique significantly differs from ours. 

Finally, among the related work, we would like to stress the result of |HK07a| . 
which provides a systematic characterization of counterexample generation in terms of 
shortest paths problems. We use this result to generate counterexamples for the acyclic 
Markov Chains. 

In the future we intend to implement a tool to generate our significant diagnostic 
counterexamples; a very preliminary version has already been implemented. There is 
still work to be done on improving the visualization of the witnesses, in particular, 
when a witness captures a large strongly connected component. Another direction is 
to investigate how this work can be extended to timed systems, either modeled with 
continuous time Markov chains or with probabilistic timed automata. 
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Appendix: Proofs 

In this appendix we give proofs of the results that were omitted from the paper for 
space reasons. 

Observation 8.1. Let M. be a MC. Since Ac(A4) is acyclic we have a ^ Oj for every 
a G Paths* (Ac (M.)) and i ^ j (with the exception of absorbing states). 

Observation 8.2. Let a,u> and f be such that a ^/ u>. Then Vi : 3j : uji ~ a,j. This 
follows from a Qf cu and the inertia property. 

Lemma 8.3. Let M be a MC, and ats G Paths* (Ac (M)). Additionally let A ats = 



{ptail(7r)|p G GenTorr(crt),7r G Paths*(SCC^, t, {s})}. Then A ats = GenTorr(cris). 



( 3 ) Let poPi ' ' " Pk £ GenTorr(cris) and n t the lowest subindex of p such that p nt = t. 
Take p = poPi ■ ■ ■ p nt an( l 11 = Pnt ' ' ' Pk (Note that poPi • ' ' Pk = ptail(7r)). In order to 
prove that popi ■ ■ ■ pu G A ats we need to prove that 

(1) p G GenTorr(crt), and 

(2) 7T G Paths*(SCC+ t,{s}). 

(1) Let / be such that ats ^ / popi ■ ■ ■ pk and f(\ats\ — l) = k. Take g : {0, 1, . . . , \at\ — 
1} — > N be the restriction of / . It is easy to check that at ^< g p. Additionally 
/(|ert| — 1) = n t (otherwise / would not satisfy the freshness property for i = 
\at\ — 1). Then, by definition of g, we have <?(|erf| — 1) = n t . 

(2) It is clear that tt is a path from t to s. Therefore we only have to show that every 
state of tt is in SCC t + . By definition of SCC t + , tt = t G SCC^ and s G SCC t + 
since s G Out gcc + . Additionally, since / satisfies inertia property we have that 

Vf(\<rt\-i)<j<f(\o-ts\-i) ■ Pf(\at\-i) ~ Pj, since f{\at\ - 1) = n t and tt = p nt ■ ■ ■ p k we 
have Vo< J <| 7r |-i : t ~ iTj proving that nj G SCC/" for j G {1, • • • , \ir\ — 2}. 

( C ) Take p G GenTorr(<rt) and tail(7r) G Paths* (SCCf , t, {s}). In order to prove that 
ptail(7r) G GenTorr(cris) we need to show that there exists a function g such that: 



(1) ats ^ g ptail(7r), 

(2) fl (|(rt«|-l) = |ptail(7r)|-l. 

Since p G GenTorr(crf) we know that there exists / be such that at <f p and 
f(\at\ - 1) = \p\ - 1. Wc define g : {0, 1, . . ., |crts| - 1} -> {0, 1, . . . , |/otail(7r)| - 1} by 



(1) It is easy to check that ats C g ptail(7r). Now we will show that g satisfies Freshness 
and Inertia properties. 

Freshness property: We need to show that for all < i < \ats\ we have Vo<j<g(-i) : 
ptail(n) g (i) ptail(n)j. For the cases i G {0, . . . , \at\ — 1} this holds since at <f p 
and definition of g. 

Consider i = \crts\ — l, in this case we have to prove Vq<j< |p taii(7r) | — i : pt a il( 7r )|ptaii(7r)| 
ptail(7r)j or equivalently V <j<|ptaii(ir)|-x : s / ptail^Tr)^ 
Case j G {\p\, . . . |/0 tail(7r) — 1} 

Since n G Paths* (SCC^~, t, {s}) and s G Outg CC + we have Vo<y<| taii(-7r)|-i : s 7^ 

tail(7r)j 
Case j G {0, . . . , |p| - 1} 

Since ats G Paths* (Ac(.M)) and Observation ^ . 1 1 we have V <j< |o-t| i : s 7^ &bj. 

Additionally, at -<f p, def. g, and Observation 18.21 imply Vo<j<ipi : s 7^ pj or 
equivalently V <j<| p | : s 9^ ptail(7r)j. 



Proof. . 
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Inertia property: Since it G Paths* (SCC^~ , t, {s}) we know that Vo<y<M-i : t ~ nj 
which implies that V| p |_x<j<utaii(7r)|-i : tadl(7r)|pi_i ~ ptail(7r)j or equivalcntly 
^g(\a\-i)<j<g(\(xs\-i) '■ P^K 71 ) g(\p\-i) ~ ptail(7r)j showing that g satisfies the iner- 
tia property. 
(2) Follows from the definition of g. 

□ 

Theorem l5.10i Let M = (S, sq,V, L) ieaMC. Then for every rail a G Paths* (Ac(M)) 
we have 

Pr ((a)) = Pr, (Torr(cr)). 
Proof. By induction on the structure of a. 

Base Case: Pr Ac( _ M) ((s )) = Pr Ac( _ M) (Paths(Ac(X), s )) = 1 = Pr_ M (Paths(A^, s )) = 
Pr_ M (Torr(s )). 

Inductive Step: Let i be such that last(cr) = t. Suppose that t G S G om- Then 
P r Ac(M)((°' s )) 

= Pr M (Toir(a)) -V(t,s) 

{Inductive Hypothesis and definition of V} 
= P^(l±J peGc „To rrW ^)) ' ^' S ) {Lem.153 

= EpeGo„Torr( CT ) Pl A1«P)) ' Pr M ((t«)) 
= EpeGenTorrCa) Pr .M «P tail(tfl)» 

{Distributivity and last (/a) = i for all p G GenTorr(a)} 

= Ep£GcnTorr(>),7r6Paths(SCC+.t,{s}) P r 7W ( (P t^K 71 ") ) ) 

= E P6 ^.^«P)) {Dfn. Z\} 

= EpeGenTorr(crs) P 'M 

((/?)) {Lem. mm- 

= P r ^ (WpeGenTorrfcrs) W) 

= Pr (Torr(crs)) {Lem. [57 



Now suppose that i G Sinp- We denote by Ac('P) to the probability matrix of 
kc(M), then 

= Pr Ao(A1) ((cr)).Ac(P)(t, S ) 

= Pr A1 (Torr(a))-Ac(P)(M) {HI} 
= P ^(l±J pe GenTo ir( .)(p)) • Ac(P)(t, a) {Lem. El} 

= (Ep eG enTo r r( CT )Pr M ((p))) -Ac^X*,*) 

= E„ £ GenT OT r W P^ «P» ' P^,, (Paths(SCC+ , t, {*})) 

{By definition of Ac('P) and distributivity} 

= Ep6GonTorr(<r) P r Ai((p)) ' E^Paths* (SCC+ ,*,{«}) P^.t^ 71 /) 

= E pe Gc„Torr( ( T). 7r ePaths*(SCC t + ,t,{ S }) Pl "^ ( (P tail M ) ) {Dfa. ^} 

= EpeGcnTorr^s) Pl ".M «P» { Lc!m - I 

= P^^pGGcnTor^crs)^)) 

= Pr jM (Torr(as)) {Lem.[ 

□ 
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